Hierarchical based sequencing machine learning model

ABSTRACT

A hierarchical based sequencing (HBS) machine learning model. In one example embodiment, a method of employing an HBS machine learning model to predict multiple interdependent output components of an MOD output decision may include determining an order for multiple interdependent output components of an MOD output decision. The method may also include sequentially training a classifier for each component in the selected order to predict the component based on an input and based on any previous predicted component(s).

FIELD

The embodiments discussed herein are related to a hierarchical basedsequencing (HBS) machine learning model.

BACKGROUND

Machine learning is a form of artificial intelligence that is employedto allow computers to evolve behaviors based on empirical data. Machinelearning may take advantage of training examples to capturecharacteristics of interest of their unknown underlying probabilitydistribution. Training data may be seen as examples that illustraterelations between observed variables. A major focus of machine learningresearch is to automatically learn to recognize complex patterns andmake intelligent decisions based on data.

One main difficulty in machine learning lies in the fact that the set ofall possible behaviors, given all possible inputs, is too large to becovered by a set of training data. Hence, a machine learning model mustgeneralize from the training data so as to be able to produce a usefuloutput in new cases.

One example of machine learning is traditional structured prediction(SP). Traditional SP is a single model approach to dependent output.With SP, once an input feature vector x is specified, a single correctoutput vector z can be fully specified. Thus the output vector z isfully conditioned on the input feature vector x and the different outputcomponents of output vector z (z₁, z₂, . . . ) are conditionallyindependent of each other given the input feature vector x. Thus, theprobability of z₁ given x is equal to the probability of z₁ given x andz₂, or p(z₁|x)=p(z₁|x, z₂). However, traditional SP cannot handle aninterdependent relationship between different output components. Inaddition, traditional SP cannot handle a problem having multiple correctoutput decisions for a given input.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one example technology area where some embodiments describedherein may be practiced.

SUMMARY

In general, example embodiments described herein relate to methods ofemploying a hierarchical based sequencing (HBS) machine learning modelto predict multiple interdependent output components of a multipleoutput dependency (MOD) output decision. The example methods disclosedherein may be employed to solve MOD problems.

In one example embodiment, a method includes employing a machinelearning model to predict multiple interdependent output components ofan MOD output decision.

In another example embodiment, a method of employing an HBS machinelearning model to predict multiple interdependent output components ofan MOD output decision may include determining an order for multipleinterdependent output components of an MOD output decision. The methodmay also include sequentially training a classifier for each componentin the selected order to predict the component based on an input andbased on any previous predicted component(s).

In yet another example embodiment, a method of employing an HBS machinelearning model to predict multiple interdependent output components ofan MOD output decision may include selecting an order for multipleinterdependent output components of an MOD output decision. The methodmay also include training a first classifier to predict the firstcomponent in the selected order based on an input. The method mayfurther include training a second classifier to predict the secondcomponent in the selected order based on the input and based on thefirst predicted component.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 is a schematic block diagram illustrating an example leadresponse management (LRM) system including an example contact server;

FIG. 2 is a schematic block diagram illustrating additional details ofthe example contact server of FIG. 1;

FIG. 3A is a schematic flow chart diagram illustrating an examplehierarchical based sequencing (HBS) machine learning model;

FIG. 3B is a text diagram illustrating an example input feature vector;

FIG. 3C is a schematic flow chart diagram illustrating a first examplemultilayer perceptron (MLP) neural network that is employed to predict afirst interdependent output component based on the example input featurevector of FIG. 3B;

FIG. 3D is a schematic flow chart diagram illustrating a second exampleMLP neural network that is employed to predict a second interdependentoutput component based on the example input feature vector of FIG. 3Band based on the predicted first interdependent output component of FIG.3C;

FIG. 4 is a schematic flow chart diagram of an example method ofemploying an HBS machine learning model to predict multipleinterdependent output components of a multiple output dependency (MOD)output decision;

FIG. 5 is a schematic flow chart diagram of multiple correct MOD outputdecisions;

FIG. 6 illustrates an example computer screen image of a user interfaceof an example customer relationship management (CRM) system;

FIG. 7 illustrates an example computer screen image of a user interfaceof an example LRM system;

FIG. 8A illustrates an example computer screen image of an example leadadvisor display before a lead has been selected by an agent; and

FIG. 8B illustrates an example computer screen image of the example leadadvisor display of FIG. 8A after a lead has been selected by an agent.

DESCRIPTION OF EMBODIMENTS

Some embodiments described herein include methods of employing ahierarchical based sequencing (HBS) machine learning model to predictmultiple interdependent output components of a multiple outputdependency (MOD) output decision. The example methods disclosed hereinmay be employed to solve MOD problems.

As used herein, the term “multiple output dependency” or “MOD” refers toan output decision, or a problem having an output decision, thatincludes multiple output components which are interdependent in thateach component is dependent not only on an input but also on the othercomponents. Some example MOD problems include, but are not limitedto: 1) which combination of stocks to purchase to balance a mutual fundgiven current stock market conditions, 2) which combination of playersto substitute into a lineup of a sports team given the current lineup ofthe opposing team, and 3) which combination of shirt, pants, belt, andshoes to wear given the current weather conditions. In each of theseexamples, each component of the output decision depends on both theinput (current stock market conditions, an opposing team lineup, orcurrent weather conditions) and the other components (the other stockspurchased, the other substituted player, or the other clothingselected). Other examples of MOD problems may relate to hostagenegotiations, retail sales, online shopping carts, web contentmanagement systems, customer service, contract negotiations, or crisismanagement, or any other situation that requires an output decision withmultiple interdependent output components.

Another example MOD problem is lead response management (LRM). LRM isthe process of responding to leads in a manner that optimizes contact orqualification rates. Leads may come from a variety of sources including,but not limited to, a web form, a referral, and a list purchased from alead vendor. When a lead comes into an organization, the output decisionof how to respond to the lead may include multiple interdependentcomponents such as, but not limited to, who should respond to the lead,what method should be employed to respond to the lead, what contentshould be included in the response message, and when should the responsetake place. Each of these components of the output decision depends onboth the input (the lead information) and the other components. Forexample, the timing of the response may depend on the availability ofthe person selected to respond. Also, the content of the message maydepend on the method of response (e.g. since the length of an emailmessage is not limited like the length of a text message). Although theexample methods disclosed herein are generally explained in the contextof LRM, it is understood that the example methods disclosed herein maybe employed to solve any MOD problem.

Example embodiments will be explained with reference to the accompanyingdrawings.

FIG. 1 is a schematic block diagram illustrating an example LRM system100. As depicted, the example LRM system 100 includes various componentssuch as a public switched telephone network (PSTN) 110, usercommunication and/or computing devices 112, a TDM gateway 120 connectingthe PSTN 100 to an internet 130, remote agent stations 121, workstations128, a call center 140, an internet gateway 150 connecting a local areanetwork 160 to the internet 130, a web server 170, a contact server 200,a lead data server 190, local agent workstations 192, and controlworkstations 194. The various components of the example LRM system 100operably interconnected to collaboratively improve a process ofresponding to leads in a manner that optimizes contact or qualificationrates.

As disclosed in FIG. 1, the remote agent stations 121 include wirelessphones 122, wired phones 124, wireless computing devices 126, andworkstations 128. In certain embodiments, the wireless phones 122 or thewired phones 124 may be voice over internet protocol (VOIP) phones. Insome embodiments, the computing devices 126 or the workstations 128 maybe equipped with a soft phone. The remote agent stations 121 enableagents to respond to lead from remote locations similar to agentsstationed at the workstations 192 and directly connected to the localarea network 160.

In one example embodiment, the local area network 160 resides within acall center 140 that uses VoIP and other messaging services to contactusers connected to the PSTN 110 and/or the internet 130. The variousservers in the call center 140 function cooperatively to acquire leads,store lead information, analyze lead information to decide how best torespond to each lead, distribute leads to agents via agent terminalssuch as the local agent workstations 192 and the remote agent stations121 for example, facilitate communication between agents and leads viathe PSTN 110 or the internet 130 for example, track attempted andsuccessful agent interaction with leads, and store updated leadinformation.

The web server 170 may provide one or more web forms 172 to users viabrowser displayable web pages. The web forms may be displayed to theusers via a variety of communication and/or computing devices 112including phones, smart phones, tablet computers, laptop computers,desktop computers, media players, and the like that are equipped with abrowser. The web forms 172 may prompt the user for contact data such asname, title, industry, company information, address, phone number, faxnumber, email address, instant messaging address, referral information,availability information, and interest information. The web server 170may receive the lead information associated with the user in response tothe user submitting the web form and provide the lead information tocontact server 200 and the lead data server 190, for example.

The contact server 200 and the lead data server 190 may receive the leadinformation and retrieve additional data associated with the associateduser such as web analytics data, reverse lookup data, credit check data,web site data, web site rank information, do-not-call registry data,data from a customer relationship management (CRM) database, andbackground check information. The lead data server 190 may store thecollected data in a lead profile (not shown) and associate the user withan LRM plan (not shown).

The contact server 200 may contact a lead in accordance with anassociated LRM plan and deliver lead information to an agent to enablethe agent to respond to the lead in a manner that optimizes contact orqualification rates. The particular purpose of such contact orqualification may include, for example, establishing a relationship withthe lead, thanking the lead for their interest in a product, answeringquestions from the lead, informing the lead of a product or serviceoffering, selling a product or service, surveying the lead on theirneeds and preferences, and providing support to the lead. The contactserver 200 may deliver the information to the agent using a variety ofdelivery services such as email services, instant messaging services,short message services, enhanced messaging services, text messagingservices, telephony-based text-to-speech services, and multimediadelivery services. The agent terminals 121 or 192 may present the leadinformation to the agent and enable the agent to respond to the lead bycommunicating with the lead.

FIG. 2 is a schematic block diagram illustrating additional details ofthe example contact server 200 of FIG. 1. As disclosed in FIG. 2, thecontact server 200 includes a contact manager 210, a dialing module 220,a messaging module 230, a PBX module 240 and termination hardware 250.In the depicted embodiment, the contact manager includes an HBS machinelearning module 212, an LRM plan selection module 214, an agentselection module 216, and a lead data server access module 218. Althoughshown within the contact server 200, the depicted modules may residepartially or wholly on other servers such as the web server 170 and thelead data server 190 for example. The contact server 200 enables anagent to communicate with a lead in conjunction with an LRM plan.

The contact manager 210 establishes contact with users and agents andmanages contact sessions where needed. The contact manager 210 mayinitiate contact via the dialing module 220 and/or the messaging module230.

The HBS machine learning module 212 employs an HBS machine learningmodel to predict multiple interdependent output components of an MODoutput decision, according to the example methods disclosed herein. Inat least some example embodiments, the HBS machine learning module 212utilizes the lead data server access module 208 to access and analyzelead information stored on the lead data server 190 of FIG. 1. Once oneor more response decisions are predicted for a particular lead, the oneor more response decisions may be conveyed to the LRM plan selectionmodule 214.

The LRM plan selection module 214 presents and or selects one or moreLRM plans for a particular lead and/or offering. Similarly, the agentselection module 216 selects an agent, class of agent, or agent skillset that is designated in each LRM plan.

The lead data server access module 218 enables the contact manager 210to access lead information that is useful for contacting a lead. In oneembodiment, the data storage access module 218 enables the contactmanager 210 to access the lead data server 190.

The dialing module 220 establishes telephone calls including VOIPtelephone calls and PSTN calls. In one embodiment, the dialing module220 receives a unique call identifier, establishes a telephone call, andnotifies the contact manager 210 that the call has been established.Various embodiments of the dialing module 220 incorporate auxiliaryfunctions such as retrieving telephone numbers from a database,comparing telephone numbers against a restricted calling list,transferring a call, conferencing a call, monitoring a call, playingrecorded messages, detecting answering machines, recording voicemessages, and providing interactive voice response (IVR) capabilities.In some instances, the dialing module 220 directs the PBX module 240 toperform the auxiliary functions.

The messaging module 230 sends and receives messages to agents andleads. To send and receive messages, the messaging module 230 mayleverage one or more delivery or messaging services such as emailservices, instant messaging services, short message services, textmessage services, and enhanced messaging services.

The PBX module 240 connects a private phone network to the PSTN 110. Thecontact manager 210 or dialing module 220 may direct the PBX module 240to connect a line on the private phone network with a number on the PSTN110 or internet 130. In some embodiments, the PBX module 240 providessome of the auxiliary functions invoked by the dialing module 220.

The termination hardware 250 routes calls from a local network to thePSTN 110. In one embodiment, the termination hardware 250 interfaces toconventional phone terminals. In some embodiments and instances, thetermination hardware 250 provides some of the auxiliary functionsinvoked by the dialing module 220.

Having described a specific environment (an LRM system) and specificapplication (LRM) with respect to FIGS. 1 and 2, it is understood thatthis specific environment and application is only one of countlessenvironments and application in which example embodiments may beemployed. The scope of the example embodiments are not intended to belimited to any particular environment or application.

FIG. 3A is a schematic flow chart diagram illustrating an example HBSmachine learning model 300. The model 300 is configured to be employedin sequential decision making to predict multiple interdependent outputcomponents, namely z₁, z₂, z₃, and z₄, of an MOD output decision z.Although the output decision z includes four (4) components, it isunderstood that an HBS machine learning model could be employed inconnection with any output decision having two (2) or moreinterdependent components. The model 300 may be trained based onrecorded historical data so that it can make optimal (or near-optimal)decisions, especially when a decision is comprised of many variablesthat need to be determined at the same time.

Although the model 300 may be employed in any number of applications toproduce MOD output decisions, the model 300 is employed in FIG. 3A toproduce an LRM MOD output decision. In particular, the model 300 isemployed to decide for a given lead what response should be performednext in a sequence that will optimize the contact or qualification ofthe lead.

For example, the model 300 may be employed to produce an LRM MOD outputdecision z=(z₁, z₂, z₃, z₄), where z₁, z₂, z₃, and z₄ are fourcomponents of the output decision z, based on an input x. In thisexample, z₁=response agent title, z₂=response method, z₃=responsemessage type, and z₄=response timing. The input x may be an inputfeature vector that includes information about a particular lead.

It is understood that the components of response agent title, responsemethod, response message type, and response timing are only examplecomponents of an LRM MOD output decision. Other example components mayinclude, but are not limited to, agent or lead demographic profile,agent or lead histographic profile (i.e. a profile of events in the lifeof the agent or the lead which could include past interactions betweenthe agent and the lead), lead contact title (i.e. the title of aparticular contact person within a lead organization), agent or leadpsychographic profile (i.e. a profile of the psychologicalcharacteristics of the agent or the lead), agent or lead social networkprofile (i.e. the proximity of the agent to the lead in an online socialnetwork such as LinkedIn® or FaceBook® or in an offline social networksuch as the Entrepreneurs Organization®, civic clubs, fraternities, orreligions), agent or lead geographic profile (i.e. cities, states, orother geographic designations that define current and/or past locationsof the agent or the lead), response frequency (i.e. how often an agentcontacts a lead), and response persistence (i.e. how long an agentpersists in contacting a lead).

FIG. 3B is a text diagram illustrating an example input feature vectorx. The example input feature vector x of FIG. 3B includes informationabout a particular lead. In particular, the example input feature vectorx includes constant features about a lead, such as lead title and leadindustry, and interactive features related to interactions between anagent and the lead, such as previous number of dials and previousaction. The lead information provided by the example input featurevector x may be employed as input by the model 300 of FIG. 3A in orderto determine what is the next sequential response that should beperformed that will optimize the contact or qualification of the lead.

It is understood that the input features of lead source, lead title,lead industry, lead state, lead created date, lead company size, leadstatus, number of previous dials, number of previous emails, previousaction, and hours since last action are only example input features toan LRM MOD output decision. Other example input features may include,but are not limited to, response agent title, response method, responsemessage type, response timing, agent or lead demographic profile, agentor lead histographic profile, agent or lead psychographic profile, agentor lead social network profile, agent or lead geographic profile,response frequency, and response persistence. Additionally, inputfeatures could include data on current events, such as current eventsrelated to politics, economics, natural phenomena, society, and culture.It is further understood that where a particular input feature isemployed as an input to a particular LRM MOD output decision, theparticular input feature will not be included among the outputcomponents of the particular LRM MOD output decision.

As disclosed in FIG. 3A, there is a dependency among components z₁, z₂,z₃, and z₄. For example, a decision on the component z₂ (responsemethod) may have an influence on the decision for the component z₄(response timing). For example, if z₂=dial, an agent may need toconsider when a lead is available to talk on a phone (e.g. usuallyduring business hours of the time zone where the lead resides). Ifz₂=email, the agent may send the email at any time.

Therefore, in the example application of FIG. 3A, and as is the casewith other MOD output decisions, the components of z are dependent bothon an input x and on the other components of z. Thus, in this example,the probability of z₁ given x is not necessarily equal to theprobability of z₁ given x and z₂, or p(z₁|x)≠p(z₁|x, z₂). In otherwords, it cannot be decided what value a specific component of z shouldtake on without considering x and the values of the other components ofz.

The model 300 of FIG. 3A employs a base classifier. In particular, andas disclosed in FIG. 3A, the model 300 employs multilayer perceptron(“MLP”) neural networks MLP1, MLP2, MLP3, and MLP4 as base classifiers.It is understood, however, that the model 300 could alternatively employother types of base classifiers including, but not limited to, othermultilayer neural networks, decision trees, and support vector machines.

FIG. 3C is a schematic flow chart diagram illustrating the MLP neuralnetwork MLP1 that is employed to predict the first interdependent outputcomponent z₁ based on the input feature vector x of FIG. 3B. In FIG. 3C,the input feature vector x is received by an input layer of the MLPneural network MLP1 and then processed by a hidden layer and an outputlayer to predict z₁ε{z₁₁, z₁₂, z₁₃}.

FIG. 3D is a schematic flow chart diagram illustrating the MLP neuralnetwork MLP2 that is employed to predict the second interdependentoutput component z₂ based on the input feature vector x of FIG. 3B andbased on the predicted first interdependent output component z₁ of FIG.3C. In FIG. 3D, the input feature vector x and the input z₁ are receivedby an input layer of the MLP neural network MLP2 and then processed by ahidden layer and an output layer to predict z₂ε{z₂₁, z₂₂, z₂₃}.

FIG. 4 is a schematic flow chart diagram of an example method 400 ofemploying an HBS machine learning model to predict multipleinterdependent output components of an MOD output decision. The method400 may be implemented, in at least some embodiments, by the HBS machinelearning module 212 of the contact manager 210 of the contact server 210of FIG. 1. For example, the HBS machine learning module 212 may beconfigured to execute computer instructions to perform operations ofemploying the HBS machine learning model 300 of FIG. 3A to predictmultiple interdependent output components z₁, z₂, z₃, and z₄ of an MODoutput decision z, as represented by one or more of blocks 402 and 404of the method 400. Although illustrated as discrete blocks, variousblocks may be divided into additional blocks, combined into fewerblocks, or eliminated, depending on the desired implementation. Themethod 400 will now be discussed with reference to FIGS. 1-4.

The method 400 may begin at block 402, in which an order is selected formultiple interdependent output components of an MOD output decision. Forexample, the HBS machine learning module 212 may select an order for themultiple interdependent output components z₁, z₂, z₃, and z₄ of the MODoutput decision z. The MOD output decision z has four componentsincluding response agent title, response method, response message type,and response timing. One possible order that could be selected isresponse agent title, response method, response message type, andresponse timing. Another possible order may be response method, responseagent title, response message type, and response timing.

Various methods may be employed to determine the order of the outputcomponents. For example, one method to determine the order may includetrying all possible orders on testing data and then selecting the onewith the best overall performance. In the example embodiment disclosedin FIG. 3A, the order has been selected to be z₁=response agent title,z₂=response method, z₃=response message type, and z₄=response timing.

In block 404, a classifier for each component in the selected order issequentially trained to predict the component based on an input andbased on any previous predicted component(s). For example, the HBSmachine learning module 212 may sequentially train the MLP neuralnetworks MLP1, MLP2, MLP3, and MLP4 to predict the components z₁, z₂,z₃, and z₄ in the selected order based on the input feature vector x ofFIG. 3B and based on any previous predicted component(s). Thus, MLP1 istrained from (x; z₁) to predict response agent title z₁ using x asinput; MLP2 is trained from (x, z₁; z₂) to predict response method z₂using x and z₁ as input; MLP3 is trained from (x, z₁, z₂; z₃) to predictresponse message type z₃ using x, z₁, and z₂ as input; and MLP4 istrained from (x, z₁, z₂, z₃; z₄) to predict response timing z₄ using x,z₁, z₂, and z₃ as input.

In one example, assume that each component has three possible values asfollows: z₁ε{z₁₁, z₁₂, z₁₃}={sales vice president, sales manager, salesrepresentative}; z₂δ{z₂₁, z₂₂, z₂₃}={call, email, fax}; z₃ε{z₃₁, z₃₂,z₃₃}={MT1, MT2, MT3}; and z₄ε{z₄₁, z₄₂, z₄₃}={short, medium, long}. Asdisclosed in FIGS. 3A and 3C, the MLP neural network MLP1 is firsttrained to predict z₁ based on x by generating three (3) output values{p(z₁₁), p(z₁₂), p(z₁₃)} for z₁ε{z₁₁, z₁₂, z₁₃}. Next, as disclosed inFIGS. 3A and 3D, the MLP neural network MLP2 is trained to predict z₂based on both x and results of MLP1 by generating nine (3·3=9) outputvalues as follows: p(z₁₁, z₂₁)=p(z₁₁)·mlp2(x, z₁=z₁₁; z₂=z₂₁); p(z₁₁,z₂₂)=p(z₁₁)·mlp2(x, z₁=z₁₁; z₂=z₂₂); p(z₁₂, z₂₁)=_(P)(z₁₂)·mlp2(x,z₁=z₁₂; z₂=z₂₁); etc. As disclosed in FIG. 3A, the MLP neural networkMLP3 is next trained to predict z₃ based on both x and the results ofMLP1 and MLP2 by generating twenty-seven (3·3·3=27) output values, andMLP4 is trained to predict z₄ based on both x and the results of MLP1,MLP2, and MLP3 by generating eighty-one (3·3·3·3=81) output values. Theeighty-one (81) possible output decisions may then be sorted based onoutput values, and the output decision with the highest output valuesmay be chosen as the predicted output decision z.

In at least some example embodiments, the eighty-one (81) output valuesmay be scaled in order to more easily handle multiplication ofrelatively small output values. For example, logarithmic output valuesfor all eighty-one (81) possible output decisions may be calculated asfollows: log(p(x; z_(1i))·p(x, z_(1i); z₂j)·p(x, z_(1i), z_(2j);z_(3k))·p(x, z_(1i), Z_(2j), z_(3k); z_(4l))); where iε{1, 2, 3}; jε{1,2, 3}; kε{1, 2, 3}; and lε{1, 2, 3}. It is understood that calculatinglogarithmic output values is just one example of scaling output values,and other scaling techniques may be employed. It is further understoodthat the scaling of the (81) output values may be omitted in at leastsome example embodiments.

It is understood that this is but one example of sequentially training aclassifier for each component in the selected order to predict thecomponent based on an input and based on any previous predictedcomponent(s), and the block 404 is not limited to the particularapplication of this example or to the LRM MOD problem solved in thisexample.

FIG. 5 is a schematic flow chart diagram 500 of multiple correct MODoutput decisions. As disclosed in the diagram 500, the HBS machinelearning model 300 may generate multiple correct output decisions 502and 504 for a given input feature vector x. Although in a typicaldecision making process it is usually assumed that there is a uniquecorrect decision given a fixed input, for LRM MOD output decisions theremay be multiple correct decisions which may all produce similarfavorable results. A decision may be chosen among multiple correctdecisions based on available resources. For example, if a particularresponse agent with response agent title z₁=“sales manager” is notavailable at a particular time, then another correct decision withresponse agent title z₁=“sales representative” may be made.

Where multiple output decisions are simultaneously considered to becorrect, the term “correct” may refer to multiple output decisions eachhaving a substantially similar output value. For example, each of theoutput decisions 502 and 504 of FIG. 5 may have an identical orsubstantially similar output value, which indicates that performingeither output decision would produce similar favorable results.Additionally or alternatively, the term “correct” may refer to multipleoutput decisions each having an output value above a predeterminedthreshold. The threshold may be predetermined to be relatively high orrelatively low, depending on the application. Although only two correctoutput decisions are disclosed in FIG. 5, it is understood that the HBSmachine learning model 300 may generate more than two correct outputdecisions.

Having described example methods of employing an HBS machine learningmodel to predict multiple interdependent output components of an MODoutput decision with respect to FIGS. 3A-5, example systems and userinterfaces that enable agents to access and implement the resultingoutput decisions will be described with respect to FIGS. 6-8B. It isunderstood that these specific systems and user interfaces are only someof countless systems and user interfaces in which example embodimentsmay be employed. The scope of the example embodiments is not intended tobe limited to any particular system or user interface.

FIG. 6 illustrates an example computer screen image of a user interface600 of an example customer relationship management (CRM) system. Theuser interface 600 includes various controls that allow an agent tomanage customer relationships and, in particular, manage leads that areprovided by the CRM system. The user interface 600 may be presented toan agent by the web server 170 on the workstations 128 or on the localagent workstations 192 of FIG. 1, for example. The agent may use theuser interface 600 to respond to leads that have been previously storedon the lead data server 190 of FIG. 1. In particular, the lead advisordisplay 800 may allow the agent to respond to leads in a manner thatoptimizes contact or qualification rates, as discussed below inconnection with FIGS. 8A and 8B.

FIG. 7 illustrates an example computer screen image of a user interface700 of an example LRM system, such as the LRM system of FIG. 1. Like theuser interface 600 of FIG. 6, the user interface 700 includes variouscontrols that allow an agent to respond to lead. The user interface 700may be presented to an agent in a similar manner as the user interface600. The user interface also includes a lead advisor display 800.

FIG. 8A illustrates an example computer screen image of the example leadadvisor display 800 before a lead has been selected by an agent and FIG.8B illustrates an example computer screen image of the example leadadvisor display 800 after a lead has been selected by an agent. Asdisclosed in FIG. 8A, the lead advisor display 800 lists five leads.Each lead includes a name 802, a likelihood of success meter 804, and alikelihood of success category indicator 806. As disclosed in FIG. 8A,the leads are listed by highest likelihood of success to lowestlikelihood of success. Upon inquiry by the agent, by mousing-over a leadwith a mouse pointer for example, the lead may expand as shown in FIG.8A for lead “Mark Littlefield.” Upon expansion, the lead may present theagent with additional options, such as a confirm button 808, a deletebutton 810, and a “more info” link 812.

Upon selection of the “more info” link 812 by the agent, by clicking onthe more info link 812 with a mouse pointer for example, the agent maybe presented with a pop-out display 814 as disclosed in FIG. 8B. Thepop-out display 814 may present the agent with an LRM plan associatedwith the lead. This LRM plan may have been generated by the examplemethods disclosed herein and may reflect the output decision with thehighest, or among the highest, output value for the lead. As disclosedin FIG. 8B, the LRM plan for the lead named “Mark Littlefield” mayinclude employing a sales manager to send an email with message type MT1in a short timeframe, which corresponds to the output decision 502 ofFIG. 5. The agent may then simply click on the pop-out display 814 tohave the lead advisor display 800 automatically generate an email to thelead with message type MT1 that will be sent by a sales managerimmediately. Alternatively, the agent may manually override the responseplan and manually perform a different response.

Therefore, the embodiments disclosed herein include methods of employingan HBS machine learning model to predict multiple interdependent outputcomponents of an MOD output decision. The example methods disclosedherein enable the prediction of each output component based on an inputand based on any previous predicted output component(s). Therefore, theexample methods disclosed herein may be employed to solve MOD problemssuch as LRM problems.

The embodiments described herein may include the use of a specialpurpose or general-purpose computer including various computer hardwareor software modules, as discussed in greater detail below.

Embodiments described herein may be implemented using computer-readablemedia for carrying or having computer-executable instructions or datastructures stored thereon. Such computer-readable media may be anyavailable media that may be accessed by a general purpose or specialpurpose computer. By way of example, and not limitation, suchcomputer-readable media may include non-transitory computer-readablestorage media including RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage or other magnetic storage devices, or anyother storage medium which may be used to carry or store desired programcode in the form of computer-executable instructions or data structuresand which may be accessed by a general purpose or special purposecomputer. Combinations of the above may also be included within thescope of computer-readable media.

Computer-executable instructions comprise, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions. Although the subject matter has been described inlanguage specific to structural features and/or methodological acts, itis to be understood that the subject matter defined in the appendedclaims is not necessarily limited to the specific features or actsdescribed above. Rather, the specific features and acts described aboveare disclosed as example forms of implementing the claims.

As used herein, the term “module” may refer to software objects orroutines that execute on the computing system. The different modulesdescribed herein may be implemented as objects or processes that executeon the computing system (e.g., as separate threads). While the systemand methods described herein are preferably implemented in software,implementations in hardware or a combination of software and hardwareare also possible and contemplated.

All examples and conditional language recited herein are intended forpedagogical objects to aid the reader in understanding the exampleembodiments and the concepts contributed by the inventor to furtheringthe art, and are to be construed as being without limitation to suchspecifically recited examples and conditions.

What is claimed is:
 1. A method comprising: employing a machine learningmodel to predict multiple interdependent distinctly-typed outputcomponents of a multiple output dependency (MOD) output decision, theemploying including: predicting a first one of the output componentsusing an input; predicting a second one of the output components usingthe same input and using the first one of the output components; andpredicting a third one of the output components using the same input,using the first one of the output components, and using the second oneof the output components.
 2. The method as recited in claim 1, whereinthe method results in multiple correct MOD output decisions.
 3. Themethod as recited in claim 2, wherein each of the multiple correct MODoutput decisions has a substantially similar output value.
 4. A methodof employing a hierarchical based sequencing (HBS) machine learningmodel to predict multiple interdependent output components of a multipleoutput dependency (MOD) output decision, the method comprising:selecting an order for three interdependent distinctly-typed outputcomponents of a MOD output decision; and sequentially training threeclassifiers for the three output components, respectively, in theselected order to predict the corresponding output component, whereinthe first output component is predicted using an input, and thesubsequent output components are predicted using the same input andusing all previously predicted output component(s).
 5. The method asrecited in claim 4, wherein each classifier comprises a multilayerperceptron (MLP) neural network, another multilayer neural network, adecision tree, or a support vector machine.
 6. The method as recited inclaim 4, wherein the input comprises an input feature vector having twoor more features.
 7. The method as recited in claim 6, wherein the inputfeature vector includes constant features about a lead and interactivefeatures related to interactions between an agent and the lead, thefeatures of the input feature vector including one or more of leadsource, lead title, lead industry, lead state, lead created date, leadcompany size, lead status, number of previous dials, number of previousemails, previous action, hours since last action, response agent title,response method, response message type, response timing, agent or leaddemographic profile, agent or lead histographic profile, agent or leadpsychographic profile, agent or lead social network profile, agent orlead geographic profile, response frequency, response persistence, anddata on current events.
 8. The method as recited in claim 7, wherein theMOD output decision is a lead response management (LRM) MOD outputdecision and the components include one or more of response agent title,response method, response message type, response timing, agent or leaddemographic profile, agent or lead histographic profile, lead contacttitle, agent or lead psychographic profile, agent or lead social networkprofile, agent or lead geographic profile, response frequency, andresponse persistence.
 9. The method as recited in claim 4, wherein themethod results in multiple correct MOD output decisions.
 10. The methodas recited in claim 9, wherein each of the multiple correct MOD outputdecisions has an output value above a predetermined threshold.
 11. Anon-transitory computer-readable medium storing a program that causes aprocessor to execute the method according to claim
 4. 12. A method ofemploying a hierarchical based sequencing (HBS) machine learning modelto predict multiple interdependent output components of a multipleoutput dependency (MOD) output decision, the method comprising:selecting an order for multiple interdependent distinctly-typed outputcomponents of an MOD output decision; training a first classifier topredict the first component in the selected order using an input;training a second classifier to predict the second component in theselected order using the same input and using the first predictedcomponent; and training a third classifier to predict the thirdcomponent in the selected order using the same input, using the firstpredicted component, and using the second predicted component.
 13. Themethod as recited in claim 12, further comprising: training one or moreadditional classifiers to predict one or more additional components inthe selected order using the same input and using the previouslypredicted output components.
 14. The method as recited in claim 12,where the first, second, and third classifiers each comprises amultilayer perceptron (MLP) neural network, another multilayer neuralnetwork, a decision tree, or a support vector machine.
 15. The method asrecited in claim 12, wherein the MOD output decision is a lead responsemanagement (LRM) MOD output decision and the input comprises an inputfeature vector including constant features about a lead and interactivefeatures related to interactions between an agent and the lead, thefeatures of the input feature vector including one or more of leadsource, lead title, lead industry, lead state, lead created date, leadcompany size, lead status, number of previous dials, number of previousemails, previous action, hours since last action, response agent title,response method, response message type, response timing, agent or leaddemographic profile, agent or lead histographic profile, agent or leadpsychographic profile, agent or lead social network profile, agent orlead geographic profile, response frequency, response persistence, anddata on current events.
 16. The method as recited in claim 15, whereinthe components include one or more of response agent title, responsemethod, response message type, response timing, agent or leaddemographic profile, agent or lead histographic profile, lead contacttitle, agent or lead psychographic profile, agent or lead social networkprofile, agent or lead geographic profile, response frequency, andresponse persistence.
 17. The method as recited in claim 12, wherein theMOD output decision relates to sports, hostage negotiations, retailsales, online shopping carts, web content management systems, customerservice, contract negotiations, or crisis management, or somecombination thereof.
 18. The method as recited in claim 12, wherein themethod results in multiple correct MOD output decisions.
 19. The methodas recited in claim 18, wherein each of the multiple correct MOD outputdecisions has a substantially similar output value or has an outputvalue above a predetermined threshold.
 20. A non-transitorycomputer-readable medium storing a program that causes a processor toexecute the method according to claim
 12. 21. The method as recited inclaim 4, wherein selecting the order for the at least threeinterdependent output components of the MOD output decision includesselecting the order from one of multiple different possible orders forthe at least three interdependent output components of the MOD outputdecision.
 22. A method of employing a hierarchical based sequencing(HBS) machine learning model to predict multiple interdependent outputcomponents of a multiple output dependency (MOD) output decision, themethod comprising: selecting an order for multiple interdependent outputcomponents of an MOD output decision; and sequentially training aclassifier for each of the output components in the selected order topredict the output component based on an input and based on any previouspredicted component(s), wherein the input comprises an input featurevector having two or more features, and wherein the input feature vectorincludes constant features about a lead and interactive features relatedto interactions between an agent and the lead, the features of the inputfeature vector including one or more of lead source, lead title, leadindustry, lead state, lead created date, lead company size, lead status,number of previous dials, number of previous emails, previous action,hours since last action, response agent title, response method, responsemessage type, response timing, agent or lead demographic profile, agentor lead histographic profile, agent or lead psychographic profile, agentor lead social network profile, agent or lead geographic profile,response frequency, response persistence, and data on current events.23. The method as recited in claim 22, wherein the MOD output decisionis a lead response management (LRM) MOD output decision and thecomponents include one or more of response agent title, response method,response message type, response timing, agent or lead demographicprofile, agent or lead histographic profile, lead contact title, agentor lead psychographic profile, agent or lead social network profile,agent or lead geographic profile, response frequency, and responsepersistence.
 24. A method of employing a hierarchical based sequencing(HBS) machine learning model to predict multiple interdependent outputcomponents of a multiple output dependency (MOD) output decision, themethod comprising: selecting an order for multiple interdependent outputcomponents of an MOD output decision; training a first classifier topredict the first component in the selected order based on an input; andtraining a second classifier to predict the second component in theselected order based on the input and based on the first predictedcomponent, wherein the MOD output decision is a lead response management(LRM) MOD output decision and the input comprises an input featurevector including constant features about a lead and interactive featuresrelated to interactions between an agent and the lead, the features ofthe input feature vector including one or more of lead source, leadtitle, lead industry, lead state, lead created date, lead company size,lead status, number of previous dials, number of previous emails,previous action, hours since last action, response agent title, responsemethod, response message type, response timing, agent or leaddemographic profile, agent or lead histographic profile, agent or leadpsychographic profile, agent or lead social network profile, agent orlead geographic profile, response frequency, response persistence, anddata on current events.
 25. The method as recited in claim 24, whereinthe components include one or more of response agent title, responsemethod, response message type, response timing, agent or leaddemographic profile, agent or lead histographic profile, lead contacttitle, agent or lead psychographic profile, agent or lead social networkprofile, agent or lead geographic profile, response frequency, andresponse persistence.